Imaging systems with semiconductor image sensors (e.g., charge coupled device (CCD) image sensors, complementary metal oxide semiconductor (CMOS) image sensors, etc.) may use a variety of mechanisms for autofocus of an image to be captured. These mechanisms may use, for example, a sweep through a range of the focal lengths to find an appropriate focal length that produces a sharp image. These types of mechanisms may require a relatively long time to find the appropriate focal distance and may lead to a relatively poor user experience. Other mechanisms may include using phase detection (PD) where some of the pixels of the image sensor may be devoted to identifying a focal length and/or the focal displacement. In this case, devoting some pixels of the imaging sensor may lead to reduced image quality, as well as additional image processing to remove any effects of the PD pixels from the captured image. Additionally, image sensors with relatively high pixel counts may result in a user perceptible latency in determining the focus point and/or the focus offset.